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ABSTRACT 


Requirements for test and evaluation in the weapons systen 
Bequisatlon process have become Increasingly formalized and 
management of the test and evaluation activities has become more 
centralized, particularly with the formation of the Major Range 
and Test Facility Base and establishment of the Uniform Funding 
Policy. Management of the Navy portion of the Major Range and 
Test Facility Base has been delegated to the Naval Air Systems 
Command Test and ftvaluation Group; however, most of the use of 
these racilities comes from the Systems and Engineering Group. 
Some of the problems resulting from this divided responsibility 
are discussed here and recommendations for improvements are 


presented. 
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1, INTRODUCTION 


A. STATEMENT OF THE PROBLEM 

In its role of Developing Agency for Naval aircraft weapons 
systems, including airframes, avionics equipment, the Naval Air 
Systems Command (NAVAIR) must maintain the capability to conduct 
test and evaluation (T&E) of the equipment it is developing. 
This has been accomplished through the establishment and develop- 
ment of a group of T&E field activities with differing responsi- 
bilities for the various areas of interest for which NAVAIR 
mMequires testing. The role of the T&E field activities is to 
orovide the various groups at NAVAIR Headcuarters with the 
testing services necessary to ensure that the various developmental 
programs are progressing satisfactorily and that they have a 
reasonable probability of success. Also, information must be 
provided to allow for correction of deficiencies as early as 
possible, particularly in advance of expensive oreduction 
decisions. Testing after a production decision provides 2 form 
cf quality assurance and helps to identify areas requiring 
muruner correction. 

Managing the operation of the field activities, internally 
and at the Headquarters level, in the current environment of 
weapon system acquisition is complex and difficult. Several 
issues have developed which were discussed in a meeting of the 
NAVAIR Civilian Management Board and the Middle Management 
Council at Charlottesville, Virginia, in November. 1977. Some 


of these issues are listed below. 





1. There is a lack of stated definitive "corporate" 
(NAVAIRSYSCOM) atrategy related to roles of the field activities 
in the overall Naval Aviation Program. The absence of declared 
top level strategy makes it difficult to develop a unified 
corporate plan for field activity management and has resulted 
in a lack of clarity and definition in roles. [1] 


2. There has been a succession of piecemeal efforts at 
dealing with many problems relating to field activity management 
and utilization (e.g., decentralization, distribution of civilian 
personnel, workloading, etc.) which result in operating in 2 
reactive, non-integrated, and non-systematic mode. [1 


3. There are several inconsistent pressures effecting the 
effective and efficient management of field activities in the 
area of contracting for services (e.g., OMB Circular A-75, OMB 
Circular A-109, Congressional language cn FY-1978 contractin 
policies, MAT-08 policies on the in-house prosecution of R&D). YD 


4. Issues relating to the modernity of the facilities and 
equipment of the field activities have not been adequately 
treated; maintenance and modernization of the physical plants 
and buildings are not treated in a pro-active mode. [i 


5. Uncertain institutional funding existed due to shortages, 
oriority changes and unanticipated costs. ff 


6, Unproductive redundancy appeared to exist. Fostering 
this in part, was an inadequate communicaticn orocess between 
the Naval Air Systems Command (NAVAIR) Headquarters and affected 
field activities. [1] 


{. Different goals and values existed as a result of many 
varying perceptions and influences as well as too many field 
activity bosses (structural, functional). [1] 


8. Deviations from assigned functional rcles occurred due 
in part to the ready availability of resources outside assigned 
mission/function areas. These resources and the related tasks 
were viewed as tending to change the basic reason for which the 
activity existed and for which it was created. [1] 


9. NAVAIR does not have adequate knewledge and control cf 
workload (functions and tasks) performed by field activities. 
For example: 


a. Headquarters has limited knowledge of the tctal work 
being performed by our field activities. 


b. The cavabilities cf our field activities are net 
adequately defined. 





ec. The Navy Lab Program causes problems of prioriti- 
zation and responsiveness due to the separation of administrative 
and technical management. 


d. Field activities have no clear and disciplined 
statement of roles, missions, responsibilities, and actual 
menecvLons. 


Set ors GYrrreculy tO tle responsibility /authority / 
meeoumnvcadlility to field activities for programs. 


f. Activities are performing functions for organizations 
other than NAVAIR, which interferes with our priorities. 


ge. Classification of activities as currently designated 
is confusing, and in some cases inaccurate. Example: PMTC is 
not really just a T&E activity. Ti] 


10. Functions and facilities at some field activities 
(particularly RDT&E activities) duplicate and overlap each other. 
Because of duplication of capabilities, our field activities 
compete for the same kind of work. For example Tactical software 
systems are spread among several activities causing duplication 
and waste. To further compound the problem the Navy Lab Program 
tends to duplicate SYSCOM and other field efforts. [1] 


ll. Management policies, systems and procedures are inadequate. 


a. The separation of administrative and technical 
management in different groups within NAVAIR causes problems of 
praoritization and responsiveness. 


b. There is inadequate coordination within HO of field 
workload, funding and manpower resources and tasks. 


ec. There is a critical imbalance between resources and 
tasks, 


d. We have no system to adjust ceilings at field 
activities or between field activities and Headquarters. 


e. We have no system to balance facility requirements 
and workload at field activities. 


f. NAVAIR has an inadequate workload control system 
and no means to discipline that system. 


ge. NAVAIR field management policies are unclear. 


h. Headquarters provides field activities with inade- 
Quate guidance. 


i. Our employees have set up informal, inappropriate 
networks to allocate tasks to field activities. [J] 
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some of the issues above are overlapping; however, it should 
be clear that a myriad of areas exist where change is reauired. 
Some of these are spoken of in the past tense and policies have 
changed since November 1977, which would affect the management 
of the field activities, but most of the issues still exist. 
In addition to the issues cited by the Civilian Management Board 
and the Middle Management Council there are concerns relative to 
the roles of the T&E field activities and their relationships 
with the Operational Test and Evaluation Force (OPTEVFOR), 
problems associated with the designation of most of the T&E 
field activities as components of the Major Range and Test 
Facilities Base, and problems in the application of the Uniform 


Funding Policy to T&E field activity operation. 


B. OBJECTIVES 

The objectives of this thesis are to identify areas in the 
management of the NAVALR T&E field activities where improvements 
are needed and to recommend changes to improve the system. In 
Che discussion that follows, these areas of concern will be 
treated separately followed by recommendations for improvement 
in each area. Recommendations are briefly restated in the 
Summary of Recommendations. For the reader familiar with the 
areas discussed, the recommendations are identified separately 


so that the detailed discussion may be avoided if desired. 


Co METHODS 
To accomplish the objectives stated above, it is first 


necessary to examine the environment in which NAVAIR and its 


Jal 





T&E field activities must operate. Policies and procedures 
that have been established for the management of the T&E field 
activities must be discussed. Finally, the devices which have 
been developed to aid in the management must be analyzed. 

To determine the environment in which NAVAIR operates, 
eeectitically with regard to T&E and fleld activity operations, 
a study of various publications was conducted. This was followed 
by an examination of the instructions and directives of NAVAIR, 
the Naval Material Command, and DoD which establish the policies 
and procedures for NAVAIR management of its T&E field activities. 
From this, a list of questions was generated to determine how 
closely actual operations adhered to the established policies 
and procedures. Responses to these questions were requested from 
personnel at NAVAIR Headquarters and the T&E field activities. 
It is important to note that this was not a survey because the 
gGuestions did not lend themselves to simple multiple choice 
answers and only the few people closely related to the problem 
were asked to respond. Also, the questions were not uniformly 
applicable to all the respondents. The results of these efforts 
were analyzed and interpreted to allow the objectives of the 


thesis to be satisfied. 


D,. BACKGROUND 

Although problems still exist, defense systems acquisition 
has improved greatly in the 1970's. Prior to this time emphasis 
was on total package procurement; however, about 1970 it was 
realized, particularly by the Blue Ribbon Defense Panel, that 


this concept was not working. This was largely due to over 
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optimistic cost and performance estimates and a lack of compe- 
tition throughout the acquisition process. Competition existed 
only at the paper study stage but design and price competition 
were nearly non-existent. As a result of the recommendations of 
the Blue Ribbon Defense Panel and other groups, policies began 
to be developed which stressed demonstrated performance as a 
requirement for continued development. This was further 
emphasized in 1976 when Office of Management and Budget Circular 
A-109 stressed that “each agency acquiring major systems should 
provide strong checks and balances by ensuring adequate system 
test and evaluation." [2] 

The policies of A-109 are implemented in the Department of 
Defense (DOD) by DOD Directive 5000.1 supported by 5000.2 and 
5000.3, which is devoted to test and evaluation. DODD 5000.1 
mica Les ; 

Programs should be structured and resources allocated 
to ensure that the successful demonstration of.program 
objectives is the pacing activity. [3] 
Since demonstration of program objectives will require testing 
it is obvious that test and evaluation in the allocation ofr 
defense resources has increased greatly in importance. Further 
evidence of this is the participation in all Defense Systems 
Acquisition Review Councils (DSARC) of the Deputy Undersecretary 
of Defense for Research and Engineering (Test and Evaluation) 
whose responsibility it is to report to the DSARC and to the 
Secretary of Defense on test planning and results. 
The increased importance of test and evaluation has not been 


Meenoucercs problems. in its report the Blue Ribbon Defense 
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Panel recommended that each service establish an operational 

test and evaluation organization independent of both the system 
developers and the users. ‘This has resulted in the establishment 
of the Operational Test and Evaluation Agency (OTEA) by the Army 
and of the Air Force Test and Evaluation Center (AFTEC) by the 
Air Force. The Navy already had its Operational Test and Evalua- 
tion Force (OPTEVFOR). All report directly to the office of 
chief of staff of their services, assuring them of independence 
from the developers and the users. 

Over the past 10 years changes have been made in the acquisi- 
tion process which reduce concurrency and program risk. The 
price has been additional management reviews, an increase in 
required test and evaluation, more use of competitive prototyping; 
all of which greatly increase acquisition cycle time and cost. 
Gansler has cited GJ the effect in that older systems such as 
the NIKE AJAX and the HAWK were fielded in 6 years and 5 years 
respectively; whereas, newer systems such as PATRIOT and AEGIS 
will take 19 and 18 years. This increase in acquisition cycle 
meme, and the ensuing efforts to reduce it, will result in 
continual pressure to minimize the time spent on T&E and will 
directly impact the relationship between NAVAIR and its T&E 


field activities. 
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Il. THE ENVIRONMENT OF 
NAVAIR AND iTS T&E FIELD ACTIVITIES 


A. DEVELOPMENTAL VERSUS OPERATIONAL TEST AND EVALUATION 

With the formation of the operational test organizations, 
test and evaluation is now formally divided into two distinct 
areas, Developmental Test and Evaluation (DT&E) and Operational 
Test and Evaluation (OT&E). DT&E is that which is conducted by 
or for the Developing Agency. Its first purpose is as an 
integral part of the iterative development process. Testing is 
conducted to evaluate equipment design candidates and the results 
are used to modify the design. This is followed by further 
besting and modification. This process continues until the 
initial paper design has been converted to fully developed 
hardware. The second important function of DT&E is to provide 
information on the progress of the system development. Performance 
is measured against a set of time-phased cost and performance goals 
and objectives to determine if the system costs and performance 
are at the levels they should be at a given point in the pregram. 
The results can be used as the basis for continuation or adjustment 
to the development program as necessary. A third function ofr 
DT&E is to help assure the project manager that he has a reasonable 
expectation of succeeding in the OT&E which follows. 

Early OT&E also aids in the design process. To the extent 
that it is conducted on developmental hardware it helps to 
iemerrysOperatlonal deficiencies to allow for correction during 


the development process. It also allows the comparison of 
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system performance with goals and objectives to determine program 
status. The difference between OT&E and DT&E lies in the orien- 
tation towards operational as opposed to technical factors and 

the emphasis on predicting future performance rather than evalu- 
ating current performance. The primary purpose of T&E then, is 

to provide answers to the questions, "What is the program status?" 
and "What is the likely outcome of the program?" [5] 

An important program consideration is the amount of scarce 
resources to be allocated to the T&E process. Since both time 
and money are consumed by the T&E process without visibly advancing 
the program, the temptation is to minimize T&E. The danger lies 
in fielding a system which is not technically or operationally 
suitable. 

The problem is compounded by the fact that it is often 
difficult to categorize tests as operational or developmental. 
Policies have been established which discourage joint testing 
unless it is dictated overwhelmingly by the costs of conducting 
separate developmental and operational tests. The tendency, 
therefore, is towards duplication of testing. Since the orera- 
tional test agencies have no obligation to minimize overall 
program costs or maintain the program schedule, there might vce 
a tendency to over test and to be over critical in their evalu- 
ation of the results to aggrandize their own position in the 
meqguisition process. In the case of developmental testing the 
Situation is similar but slightly better. The developmental 
testers work for the program manager and are dependent on him 


for their funding; therefore, he has discretion in the amount of 
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funds and time allocated to developmental testing. The dilemma 
of the program manager is to decide how much time and money to 
spend on developmental testing. The temptation is to over- 
emphasize operational testing since he has little discretion in 
this area, thereby minimizing his testing costs. The danger in 
that approach is that a system that has not had the benefit of 
sufficient test and modification of the iterative developmental 
testing process may be killed by the sometimes overzealous 
operational testers. 

The culmination of OT&E is the OPEVAL which must be completed 
fPaaor tO approval for full scale production. This is preceded 
by a Certification of Readiness for OPEVAL which must be provided 
to ensure that the system to be tested is sufficiently represen- 
tative of the system that will ultimately be deployed. fFrocedures 
for the Certification are provided by NAVAIRINST 3960.2A and 
include a number of actions required by the various NAVAIR com- 
ponents under the overall coordination of AIR-05. In theory the 
Process is relatively straightforward. In practice it has been 
a significant problem area. Problems have arisen in determining 
that the systems proposed for OPEVAL are representative of 
production systems. Because the DT&E inevitably points up 
deficiencies which must be corrected in production, major 
differences exist between the full scale develooment models and 
the production items. Correction of these deficiencies and 
configuration differences prior to OPEVAL is often time consuming 
and costly. <A further requirement for OPEVAL is that logistic 


support, manning, and training for the operation of the system 


Mie 
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during OPEVAL be representative of fleet deployment. This is 

often not possible without considerable delay because development 
of support equipment, training of personnel, and availability of 
spares in the supply system lag significantly behind system 
development. Readiness for OPEVAL, therefore, becomes a process 

of negotiation ratner than a check-off of tne elements as intended. 
This is a delicate process for tne program manager, since he must 
balance program schedule and cost constraints against tne risks 

of failing to satisfy OPEVAL requirements for lack of some of 

the items mentioned above. 

One of the primary objectives of OT&E, and particularly the 
OPEVAL, is that it be conducted under conditions representative 
of those to be expected in field operations. The reasons for. 
Mis are Obvious; however, in practice it is often difficult to 
achieve for the following reasons: 

Ll. Inadequate threat environment, 

The operating conditions include the enemy weapons 
systems and their operational doctrine against which the system 
under test must operate. Simulations of these systems do exist, 
adequate for developmental testing; however, large scale networks 
adequate to simulate the complete operational environment are 
extremely expensive and do not exist. 

2. Insufficient numbers of test systems. 

For the same reasons it is desirable to operate against 
a large scale threat environment, it is also desirable to operate 
the system under test in conjunction with all the other systems 
it is expected to live with and in the manner it is expected to 
be operated. This is also extremely expensive and not generally 
done except for individual compatability tests. 

3. Inadequate training of personnel. 

The objective is to conduct the test utilizing personnel 


that will operate the system in the field. Since schools have 
usually not been established prior to a production decision, 





adequately trained personnel do not exist. The systems must be 
maintained by persons of higher skill level than the operators, 
usually contractor personnel. 
4, System immaturity. 

In today's environment emphasis is placed on reliability. 
The systems tested; however, have been built by different tech- 
niques than will be used in full scale production. They will 
suffer from "infant mortality" problems which the mature system 
will not have. Also, with the limited number of systems and 
operating hours available, a statistically valid estimate of 
reliability usually cannot be determined. 


See back OL production configured support equipment and 
Ppoolications. 


This is related to the maintenance personnel problem. 
The support equipment designs and the publications cannot be 
determined until the production configuration of the prime 
equipment is fairly well defined. Therefore, they are usually 
not ready for testing in time and modified general purpose test 
equipment must be used, 

For obvious reasons it is desirable that objectivity and 
independence be maximized in the test program. This is particu- 
lerwiy true of the OT&E. Since the DT&E is conducted under the 
direction of the Developing Agency, biases are almost certain to 
be present. Evaluation of the results is often difficult under 
these conditions. In the OT&E the danger exists that "nice to 
have’ features will be interpreted as necessary since OT&E is 
generally conducted without regard to system cost. Another less 
likely possibility is to minimize deficiencies in seeking early 
@eplieyment. Factors which alleviate this problem are the checks 
and balances between the DT&E and OT&E, 

Previously, emphasis on OT&E was at program stages just 
prior to the production decision because of the large funding 


commitment at that point. Lately it has been realized that 


emphasis on earlier OT&E is needed for four reasons. 
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1. In the early stages of system concept formulation it is 
necessary to know that the concepts are tactically sound. 


2. lo have any impact on system development OT&E must be 
eonducted earlier than just prior to the production decision 
Since by then most of the research and development resources 
have been expended. 


3. Cost and schedule impact can be minimized by early 
detection and correction of operational deficiencies. 


4, Commitment of funds for limited production prior to the 
major production decision requires earlier availability of test 
results. [5 


The role of OPTEVFOR is defined in Reference 6. It directs 
OPTEVFOR involvement in all phases of the acquisition cycle and 
provides for close coordination between OPTEVFOR and the developing 


agency. specifically, during the program validation phase OPTEVFOR 
well: 


Establish liaison with the developing agency for 
providing operational planning assistance in determining 
the testing required for the current project, anticipated 
future testing requirements, and defining critical opera- 
tional issues. [6] 


Participate in test planning and independently observe 
selected tests and demonstrations conducted by the develop- 
ing agency, and examine test results. ---submit an assess- 
ment of this initial test and evaluation to the Chief of 
Naval Operations, and the developing agency, including 
comments and recommendations concerning future operational 
suitability of the system, progress to date, and operational 
issues requiring further examination. 


During the full scale engineering development phase OPTEVFOR 
fal 1 : 
Independently observe selected developmental tests 
and view the data from an operational viewpoint, to 
verify readiness for operational testing. [6] 
From this it can be seen that much of the work of the T&E 
field activities will be evaluated by OPTEVFOR; therefore, it is 


necessary to be continually aware of their opinions @nd priorities. 
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Test programs may have to be adjusted to help satisfy OPTEVFOR 
requirements. Since ultimately responsibility for responding to 
criticisms of the test programs will fall to the program manager 
and his team, they should closely monitor the planning, conduct, 
ema results of the tests conducted by the field activities. This 


will be discussed further ina later section. 


B. THE MAJOR RANGE AND TEST FACILITY BASE 
In 1970 the Blue Ribbon Defense Panel recognized the need 
for better test and evaluation management and recommended a major 
DOD-wide study of test and evaluation facilities. The study was 
completed in August 1971 and resulted in the formation of the 
Major Range and Test Facility Base (MRTFB), which consisted of 
26 TXKE activities of the Army, Navy, and Air Force with a combined 
annual budget in excess of $1 billion. [7] The activities 
currently comprising the MRTFB are listed in Appendix A. 
Policies for the use, management and operation of MRTFB 
activities are established in Reference 8. The mission of the 
MRITB is to: 
Provide a broad range and test support base to all DOD 
Components responsible for development, test, evaluation 
and operation, as applicable, of material and weapon systems, 
and to other Federal Government Agencies having need for 
feao SUDPOrt. 
Overall responsibility within DOD is assigned to the Deputy 
Undersecretary of Defense for Research and Engineering, Test 
and Evaluation (DD(T&E)). Authorities and responsibilities for 
DD(T&E), Secretaries of the military departments, Facility 


commanders, and Facility users are established. Reference Q 
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assigns responsibility for management of Navy portions of the 
MRTFB to the NAVAIR Assistant Commander for Test and Evaluation 
(AIR-06). 


C. THE UNIFORM FUNDING POLICY 

Prior to fiscal year 1975 no single policy existed for 
assessing the costs of operation of test facilities. Some 
activities were funded institutionally. That is, money for 
their operations was provided from a separate appropriation 
independent of any program utilizing the test facilities. A 
Second method, industrial funding, required that facility users 
may all the costs, including overhead, of operating a facility. 
A third policy required that users pay only the direct costs 
associated with their tests, with the remainder of costs to be 
Paid trom institutional funds. There were also combinations of 
these methods which sometimes varied depending on who was the 
user. 

The study completed in August 1971, noted that because of 
the differences in funding policy, test site selection was often 
based on cost to the user instead of test site capabilities. As 
a result, in January 1973, the Deputy Secretary of Defense 
directed the establishment of a Uniform Funding Policy at 18 of 
the 26 MRTFB activities, with users paying direct costs and 
indirect costs being paid from institutional funds. The policy 
Was incorporated into DODD 3200.11 and went into effect in 
fiscal year 1975. The Navy activities to which the Uniform 


Funding Policy applies are the Naval Air Propulsion Center, 
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Naval Air Test Center, Naval Weapons Center, Pacific Missile 
Test Center, and the Atlantic Undersea Test and Evaluation 
Center. 

DODD 3200.11 states that the intent of the Uniform Funding 
moaicy as follows: 

Wmeeerovidge for interservice compatability, efficiency end 
equity, without influencing technical decisions on testing, nor 
inhibiting legitimate and valid testing. 


2. Reimbursement for use of the ranges and test facilities 
will be required. 


3. All the costs of the range or test facilities which are 
not reimbursed by users will be funded by the management agency 
See cuie particular range or facility. 


4. All DOD Components and other Federal Government agencies 
--- will reimburse the ranges and test facilities for direct 
costs, --- excluding military labor. 


5. Direct Costs. Those expenses which can be immediately 
pgedirectly identified with a specific User program (usually 
documented by a job order). These costs include direct labor, 
Girect material, minor construction, special purpose equipment 
and other like costs. They include all such expenses that can 
with reasonable effort be identified consistently and uniformly 
to specific User programs. The direct cost for labor includes 
a load factor covering items such as leave and contributions to 
the retirement program. 

immconcepu tne Uniform Funding Policy is”relatively simple. 
The implementation, however, has led to problems. Historically 
it has been difficult to predict the workload of test activities 
with any accuracy, even for the following fiscal year. Without 
an accurate estimate of direct funding it is impossible to 
accurately budget the institutional funds required. Expenditures 


for civilian personnel are relatively fixed and oricrity 


t+ty 


Or use 
of institutional funds must be in this area. The impact is on 
capital expenditures for new test capabilities. An orderly 


Preegram 1S difficult to establish because funds must be held in 
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mecrow against the possible shortfall of direct funding. This 
will be discussed in a later section. 

Froolems also result for the users of the test facilities 
because of program delays. If the user plans a test for a given 
fiscal year, allocates the money to the test activity, and because 
of system problems the test is delayed beyond the end of the 
fiscal year, the program manager finds himself at the end of 
the fiscal year with the allocated test funds he can no longer 
meeeandad ne still must provide test funding out of the following 
year's budget. 

increases in administrative workload as a result of the 
implementation of the Uniform Funding Policy have been docu- 
mented. if) These increases were due largely to the increased 
budgeting requirements and the need for more detailed estimates 
and reports. ‘The increased workload has largely been absorbed 
with existing, already limited, personnel resources. The increased 
workload was not limited to comptroller personnel, but also 
affected technical personnel and detracted from their attention 


to their technical duties. 


me Lae BUDGET PROCESS 
ieeoescuss ion 
The process by which NAVAIR must prepare budgets for 
meee ttela activity operations is complicated by the fact that 
they are funded from multiple appropriations. Institutional 
funding is provided from the Category V, Management and Support, 


funds. User funding may come from Research, Development, Test 
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and Evaluation, Navy (RDT&EN) funds or from one of the procurement 
accounts, depending on the actual project being prosecuted. The 
problem, then, becomes one of balancing the amounts budgeted from 
Pachmappropriation to avoid violation of Section 3678 of the U.S. 
Code which states that funds must only be spent for the purpose 
for which they are appropriated. Reference 10 states: 
Apportionment of funds includes an amount for unfunded 

reimbursements expected to be earned during the fiscal 

year. The allocation of these anticipated reimbursements 

must not exceed the expected year-end gross unfunded 

accounts receivable. --- Work supported by reimbursable 

orders is to be financed within the subhead under which 

the order is accepted. 

Budget estimates for Navy procurement appropriations are 
submitted by NAVAIR to NAVMAT where they are consolidated and 
submitted to the Office of Budget and Reports/Fiscal Management 
Division which is responsible for formulation of the total Navy 
budget. Budget estimates for the RDT&EN appropriation are 
submitted by NAVAIR, usually by a different office than for the 
procurement appropriations, to the Deputy Chief of Naval Material 
for Development where they are coordinated and submitted to the 
Office of Naval Research for consolidation and ultimate submission 
to the Office of Budget and Reports/Fiscal Management Division. 

The separation of budget responsibilities makes consolidation 
of the various appropriations budgets into a single budget 
Pecamace for T&E field activity operations difficult, if not 
mmeossible. Paragraph 7 of Reference 11 requires that NAVAIZ 
users of the T&E field activities submit written fiscal planning 


information to AIR-6103, the TXE Field Activity Support Branch 
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of AIR-06. This would help alleviate the problem; however, 
responses to questions do not indicate that the policy is being 


followed. 


Another problem in the budgeting process is the time limita- 
) tion on the availability of funds. Congress enacts RDT&XEN 
appropriations for two years but Navy policy is "to program 
RDT&E effort on an annual incremental funding basis as opposed 
to the fully funded basis on which procurement programs are 
developed." (21) The Navy Comptroller Manual states: 


| Although the RDT&EN appropriation is legally available 
Pou, obligation for 2 fiscal years, the objective is to 
use these funds during the initial year of availability. 
The budget should be formulated to reflect tnis funding 
objective. --- However, if the award of 2 planned new 
contract, or the issuance of a reimbursable order, is 
not possible until the second year of availability, --- 
the preferred action is to restructure the funding plan 
to utilize the original funding in the first year of 

| availability for urgent deficiencies elsewnere in the 

) program and to use the following year's appropriation 

: to fund new contracts or orders. [21] 


: This policy exacerbates the problem mentioned earlier in 
| reference to the Uniform Funding Policy. Program managers neea 


not lose their funds due to program slippages if the money 1s 





legally available for another year. 
2. Recommendations 
the policy requiring users® of the Tam field activities 
womswomit written fiscal planning information to AIR-6103, if 


followed, would imprcve the ability to prevare accurs3ate consoli- 





Gated budget estimates for T&: field activity cperations. 
Adherence to this policy should be further emphasizec 4s it 
would help give NAVAIR managers a better grasp of the overall 


Mesos Of TE in the acquisition process. 
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ime DOlicy which requires obligation of all of thé 
RDISEN funds of a fiscal year appropriation within th2 first 
year of availability should be reexamined. There may be more 
advantages than disacvantages in allowing the money to be 


obligated throughout the full two years it is availsble. 


E. MAJOR AND LESS-THAN-MAJOR SYSTEM ACQUISITIONS 

Policies and procedures for major systems acquisitions have 
been well established and documented at length. This area nas 
received considerable attention because of the large sums of 
money involved. For less-than-major systems the sum involved 
in a single acquisition is smaller end, therefore, of less 
importance. Nevertheless, when all of the lesse-than-major 
systems acquisitions are considered in aggregate the amounts 
invelved is not trivial. Unfortunately, for these systems the 
Orocedures have not been as well established, particularly with 
regard to establishing T&E requirements and planning to satisfy 
those requirements. 

One reason for the problem is the volatility of the acquisition 
process. DOD Directive 5000.1, the document that prescribes 
Wolicies and procedures for major system acquisition, has teen 
rewritten twice within the last five years and 1s currently 
being rewritten again. Below the level of DODD 5000.1 there are 
numerous instructiors and directives which implement the oroad 
policy guidelines. When the major policy document changes as 
rapidly as it has, these subordinate documents almost never 
Cacch up. This results in directives which are often in conflict 


with each other and it is often impossible to be in compliance 
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with ali the current instructions. Also, programs are generally 
of such duration that they outlast changes in policy. This is 
particularly true when it is realized that many programs consist 
of derivatives of existing programs and, therefore, do not fit 
into the current phases of the acquisition process. 

Policies and procedures for test and evaluation establisned 
by higher authority are implemented within NAVAIR by NAVAIRINST 
3960.2A which states: 

imemmrOovislons Of this instruction apply to the following: 


muemearcaulSition orojects including these classified ais 
Acquisition Categories (ACAT) I, II, III, and IV projects. 


Projects involving significant modifications tc existing 
systems and equipment whose Research, Development, Test and 
Evaluation (RDT&E) and Procurement Appropriation Funding 
Requirements and other criteria would gualify them as ACAT 
mee Lit, or IV projects. 


Individual subsystems, equipment or components develored 
as a part of a major system such as @2 radar equipment, a ccmputer, 
acoustic processor, etc. which are susceptible to separate T&eE 
and which are required by competent authority to be the subject 
of a separate Test and Evaluation Master Plen (TEMP) or Test and 
Evaluation Plan (TEP). 


"Off-the-shelf" commercial systems and equipments or signi- 
ficant modifications thereof when the program decision authority 
aporoves this alternative in lieu of new development to meet an 
established mission need. 

Individual subsystems, equipment or components which are not 
developed as a part of 2 major wearon system and which are 
susceptible to separate T&E and are required by competent 
authority to be the subject of a separate TEMP to TEP. [12] 

Definitions of the Acquisition Categories are given in 
Appendix B. The TEMP and TEP will be discussed ina later 
section. From the statement above and the definition of tne 
Acquisition Categories it can be seen that it is unlikely that 


the provisions of a formal T&E program wouid not apply to most 
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acguisitions. NAVAIRINST 3960.2A is relatively new and it will 
take some time to determine if the policies are followed for 311! 
meouisition projects; however, if it i8 applied, it will go a 
long way towards improving the budgeting of T&E field activity 
operations. This is especially important in today's environment 
of conversions in lieu of prccurement and service life extension 


programs. 


F. THE TEST AND EVALUATION MASTER PLAN 

To help make the T&E of major systems an orderly process, 
the program manager is required by OPNAVINST 3960.10 to submit 
a Test and Evaluation Master Plan (TEMP). The TEMP is the 
controlling T&E planning document vrepared by NAVAIR in collator- 
ation with NAVMAT, OPTEVFOR, and the Board of Inspection and 
Survey, when applicable, and is approved by OPNAV. 12 I¢% 
serves as a contract between the Developing Agency and the uger 
for the T&E to be conducted throughout the program. Included 
are informaticn on program goals and milestones, system parameters, 
Dericrmance and cost thresholds, and test resource requirements. 
The test schedule and the level of testing to be conducted are 
meved 2O the major project milestones. Separate secticns are 
provided for DT&E and OT&E. The DT&E section is prepared by 
NAVAIR and the OT&E section is prevared by OPYTEVFOR.,. Emphasis 
is placed on resolving issues prior to submittal to OFNAV;, 
however, sometimes this is not possible and the TEMP must be 


submitted with open issues for OPNAV resolution. 
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The TEP is the controlling T&E planning document for ACAT 
IV projects and is approved by the Deputy Commander, NAVAIR. 

It is prepared within NAVAIR and contains the same type of 
Planning information contained in the TEMP for larger programs. 

Prior to seeking a Milestone I decision the program manager 
is required to have the T&E plan prepared. For major vyvrograms 
it is required as an annex to the Decision Coordinating Paver, 
the document used as the basis for the Defense Systems Acguisition 
Review Council recommendation and the Secretary of Defense decisicn. 
The initial version is to avoid specifics of the test program 
mee concentrate on the broad test plan and the critical tests 
required for satisfying major project objectives. Also, it is 
to identify the total resource needs for the test program in 
terms of manpower, material, facilities and funding. The plan 
is to be revised during the life cycle to add details as they 
become known. The revision prior to the Milestone II decision 
maould contain sufficient test prcegram specifics to ensure that 
the planned testing is sufficient to demonstrate program 
requirements. 

Within NAVAIR Headquarters AIR-05, the systems and Engineering 
Group, is responsible for the technical content of the T&E plan 
and technical direction of the T&E program. AIR-06 is responsible 
for integrating the various elements, coordinating the staff 
review of dccuments, coordinating with the T& field activities 
and arranging for the required AIR-06 managed resources. 

The use of T&E planning documents has the potential to greatly 


improve the forecasting of user requirements and funding for the 
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TeE field activities end to aid in overall forecasting of field 
activity workload and requirement for institutional funding. 
The effectiveness and efficiency of the TEMP process has not 
been optimized, however, because it has not been sufficiently 
integrated with the other information systems availeble to 
Beovide a composite picture for all projects and all field 


BeecLvlties. 
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Tit. THE RELATIONSHIP OF THE ELEMENTS 
OF TKkE FIELD ACTIVITY CONTROL 


me APPLICABLE DIRECTIVES AND INSTRUCTIONS 

A list of most of the major directives and instructions 
applicable to the structure and management of NAVAIR T&E programs 
and the operation of the T&E field activities is included in 


Appendix C. 


eee VEVISIONS OF RESPONSIBILITY 
ioe scussi on 
Policies and procedures for plarning and conducting 
NAVAIR T&E programs are contained in NAVAIR Instruction 3960.2A. 
Overall project responsibility is delegated to the program 
managers (PMA) for the individual programs; however, the vrimary 
functional responsibilities for T&E are delegated to AIR-06 and 
AIR-05, the Systems and Engineering Group. Figures 1 and 2 show 
the relationship of NAVALR with external organizations and the 
relationship of the various elements within NAVAIR. CGucting 
from 3960.2A: 
The MATACQ Group (AIR-05) is line responsible for ‘he 
performance of the full set of acquisition functicns 
within the matrix project management concept except for 
those specific program areas assigned to other Headquarters 
Groups (e.g., Target Systems). Accordingly, Air-05 is 
meemoncible for coordination of the implemencation cf the 
comprehensive T&E programs and the planning, execution, 
and technical evaluation of the DT&E and PAT&E (except for 
BIS Acceptance Trials) elements thereof. The T&E Group 
(AIR-06) is the designated single point of contact for 
NAVAIR T&E program matters and as such is line responsible 


for the interpretation and implementation of all T&z 
policies and procedures. AIR-OO functions to develoo 
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and promulgate appropriate NAVAIR policies and procedures 

to provide T&E guidance and assistance to all organizational 
units within NAVAIR. AIR-06 also provides NAVAIR and Navy- 
wide management of specifically assigned NAVAIR T&XE field 
activities and Navy elements of the DOD Major Range and 

Test Facility Base (MRTFB). 


(The title of AIR-05 was changed from Material Acquisition Group 
to Systems and Engineering Group on 2 April 1979.) 

Significant responsibilities of AIR-0O5 and AIR-06 ir 
the control of field activity operations, as specified in NAVAIR 
Instruction 39600.2A are listed below. 
mer AIR-05: 


In collaboration with AIR-06---participate in the surveys 
of candidate TeE field activities---to essess their individual 
availability, capability, and applicability to accomplish the 
planned T&E programs---. 


Ensure---coordination with AIR-06 on matters of T&E 
policy, T&eE resource requirements, selection of supporting T&E 
field activities and contractors and funding estimates for 
support of the DT&E and PAT&E programs. 


Prepare and coordinate with AIR-06, the AIRTASK and 
contract work assignments for NAVAIR T&E field activities and 
Sonmuractors. 


For AIR-06: 


Provide executive manégement for all of the NAVAIR TxE 
field activities. 


Maintain working interfaces with all available T&E 
activities including Navy. Army, and Air Force elements of the 
DOD MRTFB, as well as the Navy laboratories and appropriate 
industrial concerns, in order to establish an awareness and 
understanding of the T&E resources available for selection to 
support NAVAIR IT&E programs. 


Genduce, surveys for the selection of field activisy 
sites for the conduct of specific acquisition project T&E 
programs---. 

Approve test site selections---. 


Render T&E program support and assistance to ?MA's---in 
the following: 


3D 






(ad) Expediting T&E reports from the T&E field 
acolvities. 


Provide T&E representation to the NAVAIR Change Control 
Boards to---enable planning for T&E resource support for the 
T&E of the modification programs at the AIR-06 managed field 
activities. 


Responsibilities of the T&E field activities are as follows: 


Provide assistance to AIR-05 and to OPTEVFOR and BIS 
wnen appropriate in structuring the T&E vrograms for acquisition 
projects and in planning the elements thereof. 


Provide budgetary estimates for the accomplishment of 
T&E workload scheduled for their respective activities. 


Identify T&E resources reGuired to support approved 
TXE plans (TEMP's and TEP's), 


Provide representatives for site survey teams when 
requested by NAVAIRHQ (AIR-06). 


Perform T&E directed by NAVAIR AIRTASK/Work Unit 
Assignments. 


Provide test reports on a timely basis to support tne 
project decision making process at milestone points identifled 
in the avoproved test plan. 


Advise NAVAIR (AIR-05/06) promptly of actual or 
threatened slippages in the completion of work assignments 
which may impact the scheduled key project milestones shown 
in the approved test plan and provide recommendations for 
dealing with the causes thereof. 


Advise NAVAIR (AIR-05/0E) promptly when test results 
indicate major deficiencies which will require system or 
equipment redesign to enable meeting the T&E program objectives 
Stated in the applicable TEMP or TEP or which are likely to 
result in the breaching of performance, schedule and funding 
thresholds stated in the applicable DCP/NDCP. 

Responsibility for introducing the controlling morkload 
at Navy MRTFB activities (the T&E field activities plus portions 
of other Naval Material Command activities) is assigned to AIR-06 
by NAVMAT Instruction 3960.10. As such, they are reguired to 


perform workload management and be responsive to the technical 
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guidance of workload sponsors (primarily AIR-05). Workload 
Sponsors are required to provide timely notification of workload 
assignments to activity commanders to ensure proper support. 

The responsibilities of various organizations and the flow of 
documentation required are shown in Figures 3 and 4. (11) 

Figure 3 applies to workload sponsored by NAVAIR; whereas, 
Figure 4 applies to workload sponsored by org2nizations external 
to NAVAIR. It should be noted that the preject estimate is 
broken into direct and institutional funds. To provide for 
advanced planning tne instruction requires that fiscal planning 
information for the next five fiscal years be provided to AIR-06 
by 1 May of each year. The MRTFB activities are to submit 
workload plans for all work assigned. As was mentioned in the 
Fem@rven on the Uniform Funding Policy, it is difficult to 
forecast the work or a T&E activity even one year in advance. 
Forecasting ror five years is extremely imprecise. This is 
particularly true because the forecasting is being done poy the 
T&E activity itself rather than the users. Even knowing what 
programs are scheduled for the next five years, it is often nct 
possible to predict requirements the users will generate for 
iesupport. 

In Figures 3 and 4, the portion of the project 2stimates 
Submitted by the fleld activities for institutional funds are 
used primarily for impreving facilities and equipment. Unfortun- 
aueiy. mo formalized provisions exist for tying these elements 
together into a coordinated plan for updating the facliities of 


ene tield activities separately or collectively. an Improvement 
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and Modernization (I&M) program has been established by AIR-06 

in an attempt to improve this situation. Various efforts such 

as Project 20 have required the field activities to project their 
facilities and equipment improvement needs so that planning for 
these requirements can be done in a centralized, logical manner. 
Provisions for incorporating into these projections requirements 


for general facilities improvements that are not directly tied 





to projects have been inadequate. Funding in this area has been 
meme uncertain, probably due to the fact that insufficient 
justifying information is available to secure the funds. Institu- 
tional funds for this purpose are usually what is left after all 
other needs have been satisfied and are inevitably insufficient. 
2. Recommendations 

To improve the ability to plan and budget the workload 
of the field activities, closer dialogue between the activities 
and the users iS recommended. The T&E fleld activities snould 
be considered to be and treated as a part of the NAVAIR team, 
not subject to the arm's-length relationship required with 
@omvractors. 

Nempmevide for required updating of facilities at the 
field activities, efforts should continue for a strong I&M 
program. This should include both direct, project related, and 


general requirements. 


Cae, Sap ACTIVITIES 
Management guidance at the field activities is primarily 


concerned with the preparation of aggregate information on 
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operating budgets and workload. Another area of concern is 
assignment of personnel ceilings. Project work is nominaliy 
assigned to a single point of contact at the field activity. 

In actuality, however, the primary project interface is between 
the operating personnel at the field activity and the "users" 
at Headquarters. Because of this, the operating people are 
serving two different, and often conflicting, masters. The 
degree to which this is a problem probably varies between field 
activities for a variety of reasons, but particularly the size 
of the organization and the degree to which management and 
operating people are separated. A result of the disconnect 
between the management and operating personnel is that while 
management is seeking additional resources to perform existing 
work, operating personnel are often seeking additional work for 


existing resources. 


Dee ERLORLTY SYSTEMS 

ieee OLSCusS on 

Currently, there is no overall priority system for 

NAVAIR. Field activities establish their own priorities based 
Soa intormal guidance from various sources and their own percep- 
tions of what is most important. This has the advantage of 
Pelowing them the flexibility to respond to Situations based on 
first-hand knowledge of the situation. Efficient scheduling is 
allowed without micromanagement from Headquarters. On the other 
hand, under this system, priorities tend to be more dependent 


on the dollar value of the test program than on its overall 
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urgency and value to fleet operations and National defense. 
Also, local establishment of priorities may tend to ignore idae'y 
Cycle considerations and the requirement to complete certain 
testing prior to a scheduled Milestone Decision. 
2. Recommendation 

To date, the absence of an overall priority system has 
not resulted in any major problems. This is primarily because 
no major program has been delayed significantly due to lack of 
T&E resources. As the requirements for T&E programs become more 
formalized and the length and complexity increase this situation 
can be expected to change. Reductions in resources will exacerbate 
the situation and delays will be incurred. When this happens it 
will be imperative that a single priority system is able to 
establish the order of precedence of the various programs. The 
problem has been recognized and AIR-610 has been working with 
AIR-620 and AIR-59 to develop a priority management process. 
These efforts should continue until a workable system has been 


devised and implemented. 


E. PERSONNEL REDUCTIONS 


: 
: 
: 


Because of policy changes in recent years a continuing 
pressure for reduced personnel at Headauarters and at the field 
activities has developed. The Office of Management and Budget 
in Circular A-76 stressed the contracting out of work to the 
private sector when practical. Congressional pressures ‘to reduce 
the average grade level of civilian employees has been increasing 
in recent years. In response to these pressures the level of 


review for filling of jobs, particularly at the middie and upper 


42 











management levels, has been raised. Quotas on the filling of 
vacancies have been established and several positions have been 
downgraded. The result has been that the number of positions 
occupied has decreased much more rapidly than ceiling point 
reductions would indicate. Furthermore, all of this is occurring 
in the face of increasing complexity of systems being acquired 
and T&E requirements for these systems. Since little relief of 
the situation can be expected in the near future, the only hope 
for solution is to use the remaining resources more efficiently 


through even greater centralization of controls to Headquarters. 
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IV. DOCUMENTATION AND INFORMATION SYSTEMS 


A. AIRTASKS AND WORK UNIT ASSIGNMENTS 
fe bascussion 

Project work is currently assigned to the field activity 
by Airtask. The policy and procedures for AIRTASKS are established 
by NAVAIRINST 3900.84. [23] The AIRTASK, generally no more than 
two pages in length, represents an outline of the work to be 
performed. It includes background information, detailed require- 
ments, source and disposition of project equipment and aircraft, 
reports required, and a funding estimate. The background 
information discusses events leading to the project and estab 
the need. The detailed requirements provide a list of the 
specific functions to be performed. These are usually very broad 
and provide maximum flexibility for interpretation as the project 
progresses and the perception of the details of the reauirements 
change. The source and disposition of project equipment and 
aircraft usually only states that existing assets should be used, 
that they will be provided, or that requirements should be 
coordinated with the NAVAIRSYSCOM Aircraft Custodian. The reperts 
required may vary greatly but usually include, as a minimun, a 
final report and recently, a detailed project plan to be submitted 
early in the prosecution of the project. The funding estimate 
states the amount of money to be provided for the project by 
separate correspondence, usually a work request, from Comptroller 


Go Comptroller. 
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Work Unit Assignments are subordinate elements of an 
AIRTASK for the performance of a specific task within the scope 
of the previously assigned AIRTASK. The same form (NAVAIR 3930/1) 
ts used for both AIRTASKS and Work Unit Assignments. 

To help ensure efficient utilization of resources NAVAIRINST 
3900.8A requires that all assignments involving T&E be coordinated 
with AIR-620, the T&E Projects Division of AIR-06. This is an 
important step in centralizing the control of field activity 
operations and promoting optimum use of their resources. On the 
other hand, it reduces the authority of the technical managers 
to deal directly with the field activities for work to be 
performed. For this reason the policy is followed reluctantly, 
feat all. 

The ALRTASK/Work Unit Assignment includes a section for 
detailed requirements. Quite often this section receives 
inadequate attention so that there is not a clear understanding 
between the originator and field activity personnel of what is 
required. This may be alleviated somewhat by referencing a work 
unit plan which has been generated by the field activity. This 
is satisfactory only if the person directing the work has read 
and understood the work unit plan and concurs in what it 
proposes. This is not always the case. Another method which 
may be used to promote understanding of the requirements is to 
require the field activity to submit a detailed program plan. 
Unfortunately, the instruction does not stress that this should 
be an iterative process, continuing until an agreement nas been 
reached. All of this may be time consuming and may be intolerable 
ON urgent projects. 
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2. Recommendations 
To avoid misunderstanding and to help preclude completion 
of the test program without satisfying the objectives, discussions 
should be held and proposed AIRTASKS /WORK UNITS should be reviewed 
Pertor to formal promulgation of the AIRTASK/WORK UNIT. This 
procedure should also be followed for detailed project plans 
when they are required. The discussions should include the user, 


the field activity, and AIR-620 representatives. 


B, TEST PLANS 
iPeeescussi1on 
Currently there is no standard method or format for test 
plans common to all field activities, nor is there a requirement 
to submit the test plan for review. This allows the field 
activities maximum latitude to establish procedures. This is 
particularly important in the area of flight safety where each 
activity assumes accident reporting responsibility and, therefore, 
must have the authority to establish their own safety procedures. 
2. Recommendations 
In spite of the need for maximum flexibility at the field 
activities, there is also a need for closer control by project 
personnel at headquarters. Field activity personnel are not 
always aware of important considerations which must be emphasized 
in the test program. To help ease this problem a base line format 
for test plans should be established which the field activities 
could modify to suit their own individual requirements. Test 


plans should be required by the AIRTASK/Work Unit Assignment to 
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be submitted for review. However, this provision must be used 
carefully to avoid loss of field activity flexibility and 


independence and to maintain an objective test program. 


See RePORTING THE RESULTS OF T&E 
i oLScussion 

For T&E to be of any value the results must be reported 
to the decision makers ina timely manner. Also, they must be 
reported in such a way that they may be easily and rapidly 
understood by someone who does not have a great deal of time to 
read lengthy reports. In the case of discrepant performance, 
the sooner the discrepancy is reported and action is taken to 
Eemmeet it, the less costly the correction is likely to be. MThe 
further the development or production process proceeds before a 
discrepancy is discovered, the more that must be corrected once 
iG is discovered. 

No standard procedures have been established for 
reporting the results of T&E by the various field activities. 
Reports have been late and of little value because of the 
lateness. Information that has been used has been passed through 
briefings, letters, and messages. The report has only served as 
a record of what has been accomplished and is used very little 
as the basis for decisions. Under this system, only major 
discrepancies are reported in a timely manner. Minor problems, 
which taken together may have major impact, are generally not 
reported until the test program is complete when, in some cases, 


it may be too late to correct them. 
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Procedures have not been developed at Headquarters to 
process incoming reports and make them readily available to 
anyone who needs them. They go to the code sponsoring the work 
and other codes designated by the field activity originating the 
report, but not necessarily to all offices requiring knowledge 
of the test results. No system exists that can be accessed by 
anyone needing the information to tell if a report has been 
submitted on a project, and, if so, how to locate the report. 

Field activity emphasis has been on current testing with 
reports being written and reviewed as the schedule allows. This 
is as much due to the reluctance of personnel to do the writing 
as it is due to any established policies. To overcome this 
reluctance, methods could be devised to make ~-the writing simpler 
and shorter. The use of deficiency reports is a good example of 
this. The deficiency reports are brief enough so that they can 
be easily generated with minimum detail and be modified as more 
is learned by submitting additional reports. They are submitted 
as problems are noted, not at the end of the test program. The 
use of deficiency reports, however, greatly increases the number 
of reports which must be tracked, including the actions taken 
as a result of the reports. This increases the requirement for 
a system to record all reports so that they may be accessed 
rapidly. 

2. Recommendations 

To improve the usefulness of T&E reports, standard 

procedures should be established for their submission. This 


should include a standard report format and should be based on 
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a system similar to that currently used for deficiency reports. 
Reporting requirements must be kept brief to allow timeliness. 
As a part or an overall T&eE information system, procedures should 
be established to track all reports for easy access and to track 


actions taken as a result of the reports. 


D, WORK UNIT SUMMARIES AND FIELD ACTIVITY PLANS 

Navy MRTFB activities are required by Reference 11 to submit 
Work Unit Summaries for all current and expected projects to 
AIR-06 for use in planning and preparing the MRTFB budget and in 
the management of direct workload at the MRTFB activities. 
Included are appropriation and vroject information and estimates 
of the manpower and funding requirements of the project. Also, 
included is information on the background, objectives and progress 
or the project. Reference 11 also directs that workioad sponsors 
provide written fiscal guidance to AIR-06 and the MRIFB activities. 
If the system works properly, the Work Unit Summaries should 
reflect this fiscal guidance. It has been tne author's experience 
that this is rarely the case; however, the instruction is 
relatively new and the situation may improve with time. 

Detailed pack-up information for the manpower and cost 
estimate or the Work Unit Summary is provided by the Field 
Activity Plan. The manpower estimates are broken down by 
functional area and by tyve of personnel (military or civilian). 
The cost estimates are broken intc manpower, material, flight 
hours, travel, contract, and funding to other activities. 


Savsousiy, chis information must be balanced against the fiscal 
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guidance from the workload sponsors and, theoretically, the 
fiscal guidance should reflect the cost estimate if there is to 
be any agreement on what is to be done for how much. Again, 
ogere 1S little indication that this 1s the case, but hopefully 


tne situation will improve. 


E. COMPREHENSIVE INFORMATION SYSTEM 
fe Discussion 

Many of the problems discussed to this point result from 
poor information flow. Much of this could be improved with 
revision and integration of the information systems used for T&E 
field activity management into a unified comprehensive information 
system. 

It is, therefore, worthwhile to examine the environment 
in which such an information system must operate, existing svstems. 
and the feasibility of establishing an information system which 
will allow efficient management of resources at the Headquarters 
and field activity levels. 

Currently various systems have been established at the 
field activities, primarily to manage their own internal 
resources and to provide accounting for the funding allocated 
for various projects. In terms of the overall NSVAIR organization 
these are primarily operations control systems with some lower 
level management control applications. To better control the 
test and evaluation process the TEMP is required for each major 
acquisition project. The TEMP outlines the testing required, 
the schedule, and required resources, Field Activity Plans and 


Work Unit Summaries outline the finarcial, manpower, and equipment 
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resource requirements and the approach to be taken. None of 
these documents, however, provides a complete picture of the 
capabilities, resources available, resources recuired, and 
workload of each of the field activities individually or in 
aggregate. The feasibility of establishing such a system should 
be investigated. 

The groundwork has now been established for the examina- 
tion of the environment of such a system in the NAVAIR organi- 
Zation. Size of the organization does ncect appear as a deterrent 
to establishment of an information system. In fact, just the 
opposite is true. The large and diverse nature would geem to 
dictate that an information system be established if adequate 
Dlanning of resource allocation and utilization is to be 
accomplished. 

The organizational structure of NAVAIR and its field 
activities may represent one of the most formidable barriers ‘to 
the establishment of an effective information system. The 
organization is extremely decentralized. Each of the field 
activities has its own command structure with its own motives 
which in actuality may or may not align witno the overall NAVAIR 
goals. Further the field activities report tc no single authority 
within headquarters. Nor do headquarters personnel deal with 
Single points of contact at the field activities. Different 
headquarters offices are involved devending on whether the 
matters in question are associated with the conduct of individual 
projects or overall administrative control and support. The 


headquarters personnel associated with specific project areas 
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generally deal directly with field activity personnel who will 
be working on their projects. Because of these factors an 
extreme amount of coordination and cocperation would be required 
if a usable information system is to be established. 

Several factors effect the time frame within which an 
information system would need to be constructed. NAVAIR, in its 
role as developing agency, must deal with continuous change. 
Technology is changing rapidly and continuously, but also the 
regulations regarding systems acquisition and thus the methods 
that are required to be used are also continually changing. This 
often dictates that the interface with the field activities change 
from time to time. These changes, however, are usually relatively 
minor, and the relationship between NAVAIR and its field activities 
has not changed substantially in several years. Field activity 
control can vrobably be made more efficient and effective through 
the use of an information system; hnowever, time constraints do 
not appear to be such that pressure exists for a develovment 
project that must be rushed to the exclusion of adequate planning. 

The organization in which the information system must 
operate is somewhat difficult to define. There are organizational 
segments which clearly have a large role as the prime contributors 
and users. However, there are other segments of the organization 
that will impact the operation whose participation is not clearly 
defined. The prime headquarters users of the system would be 
ATR-06; however, since a large portion of the interface between 
NAVAIR and the field activities includes the assignment of work 


by other parts of the organization, their inputs to the system 
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are also required. The relationships between these organiza- 
tional components is undergoing changes which will take some 
time to settle into focus. Until this occurs the organizaticn 
cannot be ccnsidered mature ard this must be recognized 4s a 
possible stumbling block to the establishment of a succesgful 
information system. 

NAVAIR Headquarters does not have in-house hardware or 
personnel to be utilized for the development and operation of 
the information system. Hardware and many of the personnel 
assets required exist at the field activities. Hardwar2 could 
orobably be made available on a time-sharing oasis; hncwever, 
whether or not personnel could be made available for the project 
would require some study. Also, it may be undesirable to have 
any one of the field activities established as the central 
location fcr information system data storage and analysis. A 
possible alternative is to contract the development and operation 
of the information system to an outside consulting organization. 
this is attractive since contractual mechanisms cculd be used to 
help control cost and schedule. The real problem arises in 
finding sufficient funding to adequately support the effort. 
Since the field activity control task is currently being performed 
without major difficulties being apparent at levels controlling 
the funds, a najor effort will be required to sell the benefits 
of the system. One possibility to accomplish this might be to 
cite examples of past inefficiencies and their sssoctiated costs 


which could have been avoided witn the proposed informetion 








system in operation. A side benefit of this wovld be to hein 
define the areas toward which the information system should be 
directed. 

Attitudes about information systems are, unfortunately, 
generally hostile. Too often the same data have been recguired 
to be submitted several times in different formats to satisfy 
different information systems only to have the results of 
considerable effort go unused for any significant decisions 
regarding the operation of the organization. Expectations for 
an information system would probably be low; therefore, any 
demonstration of successful operation weuld orobably come as a 
welcome surprise. One of the biggest problems is convincing 
the operations level people who make most of the inputs to the 
system that it is worth their while to spend the time and effort 
to make the inputs accurate. This could be significantly aided 
if the same information is required cnly once and if past 
information submitted may be easily reviewed and updated without 
complete resubmission. Also, it 18 necessary that they be 
convinced that the results of their efforts serve some useful 
purpose. For these reasons it is important that the managers 
who are potential users of the system be ccnvinced that it is 
worth their effort to stay involved in the development of the 
system beyond the establishment of the initial requirements. 

Assuming that in spite of the draw backs cited in the 
preceding paragraphs, it is decided that an information systen 
teometne controlling of fiald activity operatiors Is worth the 


Cmueortre it is possible to look at what might be some of tre 
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elements of the information system target. At least, the 
definition and some of the planning stages can be examined. 

Various systems have already been established at the 
field activity or operational level which may be modified or 
used intact as inputs to the information system. Some direction 
from headquarters as to the type of information required in the 
system is also required. For these reasons the parallel approach 
of Reference 14, where operational systems and planning and 
control systems are developed simultaneously, seems to be 30pro- 
priate for this information system development. Top down is 
ruled out because it would probably require more top management 
support than is available. This would be particularly true when 
attempting to integrate information from the various parts of 
the matrix organization. On the other hand, bottom up would ode 
easier to accomplish, but due to differences between the field 
activities it 1s doubtful that a system with sufficient integration 
to be useful could be attained this way. 

There is little requirement for the field activities to 
Gxenange information between themselves; therefore, there is no 
requirement for a high degree of integration at this point. They 
should, however, be able to pass the same information in the 
same formats to headquarters for further aggregation and 2nélysis. 
meumoeugaethis 1s desirable 1t does not meanmethat nothing useful 
can be gained treating the information received from each of the 
field activities by itself. For example, aircraft requirements 
for current and future projects can be analyzed for each fiele 


activity individually. On the other hand, requirements for 
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funding for new instrumentation and facilities must be compared 
Por all activities to establish priorities for ellocation of 
extremely scarce resources. 

Because of the psychological climate in which the system 
must operate, innovation and the attendant risk should be kept 
to a minimum. It is important to achieve early, if modest, 
successes to maintain and improve support for the system. One 
area where some innovation may be allowed is data base management. 
This technique offers considerable appeal in terms of establishing 
the system such that a given piece of data information needs to 
be entered into the system only once, which will make the system 
Getractive to tnose who must feed it. 

2. Recommendation 

The priorities, specific functions, and goals cf the 
projected information system require considerable in-depth 
investigation beyond the scope of this thesis. These could be 
developed using the preceding discussion and discussions with 
personnel who would be involved in the operation and use of the 
system. A study should be established involving, 28s a minimum, 
representatives of AIR-05, AIR-06, and the T&E field activities. 
High level attention must be maintained to ensure proper suvport 
for the system instituted as the result of this study. In the 
establishment of the information system a consultant familiar 
with the establishment of information systems should be hired to 
prevent many of the voitfalls which might not be seen by someone 


unfamiliar with the process of establishing an information system. 
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Emphasis in the development should be placed on ease of submission 
and updating of information and information requirements should 


be limited to those for which a valid use has been identified. 
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V. PROBLEM AREAS AND OPPORTUNITIES 


A. ACTIVITY MISSION STATEMENT OVERLAP 
ia Discussion 

There are four T&E field activitiessy listed in Table l, 
under the administrative control of AIR-06. In addition signi- 
ficant NAVAIR T&E functions are performed at the Naval Weapons 
Center and the Atlantic Undersea Test and Evaluation Center, 
although they are not NAVAIR field activities. The missions of 
the field activities are stated in NAVAIR Instruction 5451.82. 
Some areas are clearly within the mission of only one of the 
field activities. In other areas, however, there is considerable 
overlap in the stated missions. As an example, the mission 
Statements of the two largest field activities, the Naval Air 
Test Center (NATC) and the Pacific Missile Test Center (PMTC) 
are quoted below. For NATC the mission is: 


"To conduct tests and evaluation cof aircraft weapons 
systems and their components." [15] 


For PMTC the mission is: 
"To perform development test and evaluation, develop- 

menG Support, follow-on engineering, logistics and training 

support, for naval weapons, weapons systems and related 

devices; provide major range, technical and bése support 

for Fleet users and other DOD and government agencies.’ [25] 

This is only one example. There are overlavus with other 

activities also. NAVAIRINST 5451.82 also states the specific 
functions to be performed by the field activities in the accomrlish- 


ment of their missions. Here there is further overlap. 








Table 1 


NAVAL AIR SYSTEMS COMMAND 


T2ST AND EVALUATION FIELD ACTIVITIES 


Naval Weapons Evaluation Facility, Albugquergue, N.M. 
Naval Air Propulsion Center, Trenton, N.Jd. 
Naval Air Test Center, Patuxent River, MD. 


Pacific Missile Test Center, Point Mugu, C. 


The result of this cverlap is a certain amount of 
wasteful duplication of facilities and capabilities and the 
competition between field activities cited by the Civilian 
Management Board/Middle Management Council [1] and discussed in 
meee introduction. Certainly, some of the duplicaticn is 
necessary. All test ranges must have position measurement, 
communications, and data gathering and processing systems. Also, 
a certain amount of comvetition is inevitable end, in fact, 
beneficial because it helps to keep the field activities effi- 
cient and responsive to the needs of Headquarters. The key to 
maethner the duplication is wasteful or not is in the amount or 
marketing done by the field activities for work and the degree 
feavwnich the capacity of the field activities, as a group, is 
utilized. The marketing that accompanies the competition betweer 
the activities often results in waste because work is performed 
that is not really necessary. This would seem to indicate that 
capacity is not being utilized efficiently and some wasteful 


Guplication does exist. This is paradoxical because in many 
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cases the field activities claim to be overworked, underfunded, 
and understaffed, yet the inertia of the system causes them tc 
eontinue to seek new work, In fact, at different l@vels in the 
organization there exist different perceptions of what the work 
level really is. One level of management may be actively pursuing 
new projects while another level is trying to discourage them. 

2. Recommendations 

It would appear that one way to decrease the wasteful 

poesaecs Of competition is greater centralization of T&E project 
control in AIR-06 where a greater knowledge of the overall 
capabilities and workloads of the field activities exists. 
However, it is unlikely that this would work for two reasons. 
First, there are insufficient personnel in AIR-06 to assume the 
added responsibility and it is unlikely that additional personnel 
are available. Second, this is a reduction in the flexibility 
and authority of AIR-05 and would meet strcng resistance from 
that quarter. The best solution is probably merely to encourage 
greater awareness of the problem and greater cooperation between 
AIR-C5 and AIR-06 through the policies established in NAVMATINST 
3960.10 and NAVAIRINST 3960.2A. Proposals have been made for 
the establishment of a facilities and capabilities handbook which 
could be used to increase the awareness of all T&E facilities 
users, and particularly AIR-05, of the areas of expertise of the 
various T&E field activities. This would improve the situation 


and such a handbook should be published. 
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B. PLANNING AND BUDGETING FACILITIES IMPROVEMENTS 
ie Disecussaon 

The Uniform Funding Policy and NAVMATINST 3960.10 
require that facilities improvements that support a specific 
project be paid for by funds from thet project and that general 
improvements in capabilities be paid for by the MRTFB budget. 
In practice it is often difficult to place a given improvement 
in one category or the other. The result is that the "user" 
refuses to pay to improve the overall capability of the activity 
and AIR-06 refuses to pay for improvements to support a project 
and needed improvements are not accomplished. Further, there 
is no provision for facilities improvements which cannot be 
justified by an immediate application. This precludes long 
range planning and efficient modernization programs because 
accurate predictions of future projects are not available. 

Mne other difficulty with funddmse ftacilities improvements 
from the MRTFB budget, mentioned earlier, is the low priority 
of facilities improvements when compared with other demands for 
MRTFB funds such ag salaries and other operating expenses whicn 
are relatively fixed and must be paid. When user funding falls 
short of budget estimates it is the facility improvements which 
inevitably suffer. Even when user funding is edequate, MRTF3B 
funding must be held in escrow against pessible shortfalls. This 
results in hurried spending at year end instead of a balanced 


Preagram throughout the year. 
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2. Recommendaticns 
To alleviate these problems and allow facilities 

improvements to be based on a coordinated long range plan, the 
funding of facilities improvements should be exempted from the 
Uniform Funding Policy and funded entirely from MRTFB funds 
except when large dollar value improvements are being considered 
for a specific project. In this case, "user" funding might be 
used to supplement MRTFB funding. The negotiation for "user' 
funding should be between AIR-06 and the program manager. The 
field activities should not be involved. Further, the I&M 
portion of the MRTFB budget, which funds the facilities improve- 
ments, should become a separate allocation not subject to other 
uses. This would result in a reduction in the flexibility of 
the field activities to utilize the MRTFB funds as required but 
it would help to ensure that needed improvements would be 


provided. 
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VI. SUMMARY OF RECOMMENDATIONS 


The recommendations which have been made generally fall into 
one of three categories; changes in operating procedure, changes 
in budgetary procedure, and emphasis on implementing existing 
policies. These recommendations 2re summarized in the s2ctions 


that follow. 


A. OPERATING PROCEDURE RECOMMENDATIONS 

Dialogue between users and the T&E field activities should 
be increased. Field activities should be considered a2 part of 
the NAVAIR team rather than as contractors. 

A single integrated priority system for T&E field activity 
Support should be established. 

Detailed discussions should be held to review AIRTASKS /WORK 
UNITS and detailed program plans prior to their promulgation. 

A standard format should be developed for test plans énd 
they should be supnmitted for review by the work sponsor, 

Standard procedures should be developed for reporting the 
results of T&®E. These should be based on simplified requirements 
which minimize the effort involved in preparing the report, 
which would improve timeliness. A report tracking system should 
be established at Headquarters to vrovide easy access to the 
reports and status of required actions. 

A study effort shculd be instituted which would result i 


the establishment of 2a NAVAIR T&E information system. 
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A facilities and capabilities handbook should be prepared 
to improve awareness of users of areas of expertise of the 


mamous T&éE field activities. 


B. BUDGETARY PROCEDURE RECOMMENDATIONS 

Tne requirement to obligate all RDT&EN funds within the 
first year of thelr availability should be reexamined to determine 
if it causes more harm then good. 

Funding of facilities requirements, except in rare cases, 
should be paid for entirely from MRTFB funding. 

The I&M portion of the MRTIFB budget should be a separate 


allocation, not subject to other uses. 


C. POLICIES WHICH REQUIRE FURTHER EMPHASIS 
Users of T&E field activities should submit written fiscal 
planning information to AIR-6103 per NAVMATINST 3960.10. 
Biforts for a strong 1L&M program should continue. 
To minimize wasteful duplication of facilities and carabil- 
ities and competition between T&E field activities, policies 
of NAVMATINST 3960.10 and NAVAIRINST 3960.2A must be followed 


more closely. 
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APPENDIX A 


ELEMENTS OF THE MAJOR RANGE AND TEST FACILITY BASE 


NAVY 


me 


Pacific Missile Test Center 

Atlantic Undersea Test and Evaluation Center 
Naval Air Test Center 

Naval Air Propulsion Center 


Naval Weapons Center 
(T&E Portion Only) 


Atlantic Fleet Weapons Training Facility 


FORCE 

Eastern Test Range 

Space & Missile Test Center 

peaverlite Control Facility 

Arnold Engineering Development Center 
49O50th Test Wing 

Tactical Fighter Weavons Center 
Flight Test Center 

Armament Development and Test Center 


Air Defense Weapons Center 


ARMY 


Cold Regions Test Center 
Tropic Test Center 
Yuma Proving Ground 


Jefferson Proving Ground 
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White Sands Missile Range 
Kewa jJalein Missile Range 
Dugway Proving Ground 


Aberdeen Proving Ground 


NOTE: The Naval Weapcns Evaluation Facility is a NAVAIR T&=& 


Bread activity not included in the MRTFE. 
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APRPanolLxA 8 


DEFINITIONS OF ACQUISITION CATEGORIES [12] 


ACAT I: Projects designated by the Secretary of Defense 
(SECDEF) or Deputy Secretary of Defense (DEPSECDEF) having a 
nominal dollar value in excess of $75M RDT&E or $200M Production 
and such other lesser programs as may be so designated by SECDEF 
om DEPSECDEF. 


ACAT II: Projects designated by a Defense Systems Acquisition 
Review Council (DSARC) principal, SECNAV or CNO having a nominal 
dollar value in excess of $20M RDT&E or $50M Producticn and such 
other lesser programs as may be recommended by the Chief of 
Naval Material (CHNAVMAT), CP-090, OP-098 or the program sponsors 
(DCNO /DMSO ). 


ACAT Iif: Projects designated by the OPNAV sponsor, having 
a nominal dollar value in excess of $5M RDT&E or $20M Production 
and sucn other lesser programs recommended by CHNAVMAT, OP-O90, 
OP-098, or the DA. Pregrams below the ACAT III threshold will 
normally be designated ACAT III if they: 


(a) Directly affect the military characteristics of 
shivos, aircraft or other combat units; 


(bo) Require Operational Test and Evaluation (OT&E) to 
support key pregram decisions; or 


(c) Require Fleet RDT&E support. 


ACAT IV: Projects not designated as ACAT III or higher 
which meet any one of the following criteria: 


(a) Have a unit cost of more than $10,000, nct 
including management and support costs; 


sb Have a total project cost of $1M (R&D plus 
production) excluding shore based training devices and shore 
support equipment not used in direct support of deployed 
systems; 


(c) The hardware required formal approval for service 
use (ASU). 
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APPENDIX C 
DIRECTIVES AND iis tRve TIGNs 
DOD Directive 3200.11, Use, Management and Overation of 
Department of Defense Major Ranges and Test Facilities, 
June 18, 1974. 
DOD Directive 5000.1, Major System Acquisition, January 18, 1977. 


DCD Directive 5000.2, Major System Acquisition Process, January 
eG, 1977. 


Wepeoirective 5000.3, Test and Evaluation, January 1, 1973. 


eee NAVINS?T 5000.1, System Acquisition in the Department of 
the Navy. eS 


OPNAVINST 3900.25A. 


OPNAVINST 3960.10, Test and Evaluation. 


(D 
ca 
ct 


OPNAVINST 5440.47D, Mission and Functions of Operational T 
and Evaluation Force (OPTEVFOR), 5 March 1973. 


NAVMATINST 3910.15A. 
NAVMAT INST 3960.6A, Test and Evaluation. 


NAVMATINST 3960.8, Land-Based Test Site (LBTS) Selection Policy, 
1 September 1976. 


NAVMATINST 3960.10, Policy and Procedures for Workload Manzge- 
ment at the Navy Major Range and Test Facility Base (MRTFB) 
Activities, 3 February 1978. 


NAVMATINST 5450.31, Consolidation of Navy Ranges, Test 
Activities, and Target Test Resources, 29 March 1976. 


NAVAIRINST 3900.8A, AIRTASK and Work Unit Assignments to Field 
Activities, 22 March 1977. 


NAVAIRINST 3960.2A, Test and Evaluation, 11 August 1978. 


NAVALRINST 5451.82, Naval Air Systems Command (NAVAIR) Test and 
Evaluation Activities; Mission and Functions of, 16 December i975. 
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